Log Analysis and Visulization¶

import utils
import os
from os import path
import datetime
import pandas as pd

Result¶

Statistics Summary
- transaction per second
- time per block
- size
- gas

Features vs Time / Block Height (with all shards / per shard)
- staking transaction per second
- total transaction per second
- transaction per second
- time per block
- size
- gas

if __name__ == "__main__":
    
    date = datetime.date.today()-datetime.timedelta(days=1)
    name = "ostn_"+date.strftime("%Y_%m_%d")+".log"
    fig_dir = "../../graphs/test_logs/ostn/" + date.strftime("%Y_%m_%d") + "/"
    log_dir = "/home/ubuntu/jupyter/monitor/mini_explorer/data/" + name
    html_dir = "../../docs/graphs/test_logs/ostn/" + date.strftime("%Y_%m_%d") + "/"

    data = [html_dir, fig_dir]
    for directory in data:
        if not path.exists(directory):
            try:
                os.makedirs(directory)
            except:
                print("Could not make directory")
                exit(1)

    data = utils.read_data(log_dir)
    shard = utils.data_processing(data)
    
    utils.set_config({
        'ignore_printing_statistics_summary': False,
        'ignore_drawing_features_vs_time_per_shard' : True,
        'ignore_drawing_features_vs_block_per_shard' : True,
        'ignore_drawing_staking_features_vs_time_per_shard' : True,
        'ignore_drawing_staking_features_vs_block_per_shard' : True,
        'ignore_drawing_features_vs_time' : False,
        'ignore_drawing_features_vs_block' : True
    })
    
    config = utils.config
    ## get the index of the new chain started
    index = []
    for s in shard:
        i = s[s['time_per_block'] < 0].index.tolist()   
        if len(i) == 0:
            break
        i.insert(0,0)
        i.append(len(s))
        index.append(i)
    
    if len(index) == 0:
        new = pd.concat(shard)
        new.reset_index(inplace = True, drop = True)
        print("transaction for ", date, new['transactions'].sum())
        utils.visualization(new, fig_dir, html_dir, config)
    else:    
        for i in range(len(index[0])-1):
            df = []
            for s in range(len(shard)):
                df.append(shard[s].iloc[index[s][i]+1:index[s][i+1]])
            new = pd.concat(df)

            fig_path = fig_dir + "chain_" + str(i) + "/"
            if not os.path.exists(fig_path):
                os.makedirs(fig_path)

            utils.visualization(new, fig_path, html_dir, config)

Statistics summary for shard 0
==================================
Total data points: 500
             size            gas  transaction_per_second  time_per_block
mean   784.800000   12642.000000                 0.07525        7.977835
std    612.686692   49565.060541                 0.29503        0.495633
min    677.000000       0.000000                 0.00000       -3.082682
25%    677.000000       0.000000                 0.00000        8.000000
50%    677.000000       0.000000                 0.00000        8.000000
75%    677.000000       0.000000                 0.00000        8.000000
max   9474.000000  231000.000000                 1.37500        8.000000

Statistics summary for shard 1
==================================
Total data points: 791
             size  gas  transaction_per_second  time_per_block
mean   701.735777  0.0                     0.0        7.989404
std    439.659410  0.0                     0.0        0.298010
min    677.000000  0.0                     0.0       -0.381443
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9474.000000  0.0                     0.0        8.000000

Statistics summary for shard 2
==================================
Total data points: 792
             size  gas  transaction_per_second  time_per_block
mean   690.748737  0.0                     0.0        7.989425
std    309.270940  0.0                     0.0        0.297612
min    677.000000  0.0                     0.0       -0.375538
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9380.000000  0.0                     0.0        8.000000

Statistics summary for shard 3
==================================
Total data points: 792
             size  gas  transaction_per_second  time_per_block
mean   701.931818  0.0                     0.0        7.989437
std    439.377203  0.0                     0.0        0.297256
min    677.000000  0.0                     0.0       -0.365517
25%    677.000000  0.0                     0.0        8.000000
50%    677.000000  0.0                     0.0        8.000000
75%    677.000000  0.0                     0.0        8.000000
max   9474.000000  0.0                     0.0        8.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_04_28/chain_0/
==================================

HTML saved in

HTML saved in

HTML saved in

HTML saved in

HTML saved in

HTML saved in

Statistics summary for shard 0
==================================
Total data points: 2590
              size           gas  transaction_per_second  time_per_block
mean   1363.153282  1.909050e+05                0.044628       11.175684
std     850.066570  9.688892e+05                0.225832      105.007345
min     677.000000  0.000000e+00                0.000000       -4.152477
25%    1122.000000  0.000000e+00                0.000000        8.000000
50%    1128.500000  0.000000e+00                0.000000        8.000000
75%    1265.000000  2.249600e+04                0.000000        9.000000
max   18022.000000  1.064500e+07                1.375000     4029.000000

Statistics summary for shard 1
==================================
Total data points: 3567
              size  gas  transaction_per_second  time_per_block
mean    703.204093  0.0                     0.0        9.133543
std     568.456397  0.0                     0.0       47.238189
min     677.000000  0.0                     0.0       -3.150550
25%     677.000000  0.0                     0.0        8.000000
50%     679.000000  0.0                     0.0        8.000000
75%     679.000000  0.0                     0.0        8.000000
max   17573.000000  0.0                     0.0     2244.000000

Statistics summary for shard 2
==================================
Total data points: 3520
              size  gas  transaction_per_second  time_per_block
mean    700.737784  0.0                     0.0        8.322242
std     552.794395  0.0                     0.0       18.676403
min     677.000000  0.0                     0.0       -2.993018
25%     677.000000  0.0                     0.0        8.000000
50%     678.000000  0.0                     0.0        8.000000
75%     679.000000  0.0                     0.0        8.000000
max   17572.000000  0.0                     0.0     1116.000000

Statistics summary for shard 3
==================================
Total data points: 3562
              size  gas  transaction_per_second  time_per_block
mean    697.188658  0.0                     0.0        8.547010
std     491.512726  0.0                     0.0       29.506862
min     677.000000  0.0                     0.0       -2.966241
25%     678.000000  0.0                     0.0        8.000000
50%     679.000000  0.0                     0.0        8.000000
75%     680.000000  0.0                     0.0        8.000000
max   17573.000000  0.0                     0.0     1769.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_04_28/chain_1/
==================================

HTML saved in

HTML saved in